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ABSTRACT 



The thesis investigates the determinants of effective and ineffective 
’’instructorship" as perceived by students at the Naval Postgraduate School, 
Through a critical incident survey and subsequent content analysis a 
scheme which can express student perceptions of effective instructorship 
was developed. Categorization of good and bad instructorship incidents 
isolated sixteen factors which were found to be determinants of student 
perceptions. Examples of verbatim student comments are included in the 
form of a general feedback vehicle appropriate for instructor consideration. 
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I. INTRODUCTION 



As the educational goals of academic institutions become more demanding 
and time and cost considerations force accelerated programs of learning, 
increased attention has been focused on the quality of classroom instruc- 
tion. Attempts to measure the effectiveness of an instructor have placed 
a heavy emphasis on the solicitation of student opinion. The objective of 
the research reported here was to distill factors of instructorship which 
could express student perceptions of effectiveness. A survey of Naval Post- 
graduate School students conducted in January 1973 provided the data base. 
This study furthers research initiated by Professors Els ter, Githens and 
Senger at the Naval Postgraduate School in 1972.^ 

Student opinion regarding effectiveness can be useful both as a feedback 
aid to an instructor seeking to enhance his teaching and as a decision- 
making tool for administrative faculty review. This research has attempted 
to categorize student opinion into distinct common denominators of student 
perception regarding effectiveness or ineffectiveness. The primary thrust 
has been toward developing a coherent and systematic scheme for addressing 
perceived instructor effectiveness determinants which could facilitate 
feedback. Quantification, in terms of rating scales, could eventually be 
based upon these common denominators. 



^ R. S. Els ter, W. H. Githens, and J. D. Senger, Factors Leading to 
Satisfaction and Dissatisfaction with Teachers , paper presented at the 
Western Psychological Association Convention, 1972. 
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II. REVIEVJ OF LITERATURE 



A review of the literature relative to faculty evaluation reveals a 
diversity of methods utilized. Five of the most common procedures used 
in the evaluation of classroom teaching are discussed by Miller: class- 

room visitation by a qualified observer, routine monitoring of teaching 
materials and procedures, observation of special incidents relating to 

laudatory or negative aspects of an individual’s teaching, self evaluation 

2 

and student evaluation. A study conducted by Gaff and Wilson indicates 
that student rating is by far the most prevalent of the feasible procedures 
which have been used to obtain systematic evidence about teaching effec- 
tiveness.^ Miller however, notes that although some institutions of higher 
learning have experimented with student evaluation of teaching for several 
decades, numerous officials and professors have just discovered this 
dimension of evaluation and it has become the "in thing. 

A study of the many techniques utilized for student evaluation of 
teacher effectiveness acquaints the reader with a variety of evaluation 
forms which their respective authors believe to be the most effective. 

Few of these reports discuss the research involved in the development of 
the forms. The reader perceives that the vast majority of the devices 
are developed through an a priori or intuitive approach. The authors of 



o 

^ R. I. Miller, Evaluating Faculty Performance (San Francisco: Jossey- 

Bass, 1965), pp. 26-39. 

^ J. G. Gaff, R. C. Wilson, and others. The Teaching Environment: A 

Study of Optimum Working Cond itions for Effective College Teaching (Berkeley 
Center for Research and Development in Higher Education, 1971), p. 290. 

^ Miller., Op. Git . , p. 14. 
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these forms are professional educators and feel qualified to determine those 
qualities or factors which should be of primary importance in determining 
teacher effectiveness. The student evaluating the instructor is thus 
required to utilize a rating scale for each factor cited. As an example, 
Renner illustrates a form of this nature in his article, ”A Successful 
Rating Scale. Through repeated use and periodic analysis of data 
collected, successive refinements and improvements can be implemented in 
the rating form. It must be noted that these refinements again assume an 
ability, on the part of the author, to determine what constitutes an improve- 
ment. A variation of this technique allows the student to report those 
additional factors (positive or negative) considered to be of importance 
by including a f ree-response section as part of the form. These responses 
might contribute to a revision of the form if the author finds in them a 
pattern in consonance with his views. Few rating forms have been oriented 
toward the student^s perception of factors that constitute effective 
instruction. If student perceptions of instructor effectiveness are of 
value, logic would dictate that the factors which influence those perceptions 
should be identified. A definitive study of the factors most important to 
the student is not available. The primary focus of this research is to 
that end. 

The structure of the instructor evaluation forms used varies consider- 
ably although most use items that can be scored with relative ease. Items 
may be of the multiple-choice descriptive type or graphic type requiring 
the rater to check along a horizontal continuum. Sometimes the intervals 
in the graphic type are specific and carefully labeled while in other 

^ R. R. Renner, ”A Successful Rating Scale,’* Improving College and 
University Teaching , 15:12-14, 1967. 
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cases the intervals are left vague. In either instance, the rater may 
be asked to circle a number or a letter grade or to put a check mark in a 
box. Frequently a five-point scale is used (with "5** representing "superior” 
and "1" representing "poor" or "needs improvement"). Some items allow for 
only a dichotomous response: agree-disagree or yes-no. A few rating forms 

are entirely open-end, calling for short comments on such matters as 
instructor’s attitudes toward his subject and/or attitude of the students 
toward him. The following illustrations are extracted from Kent’s overview 
of these diverse devices routinely used for student evaluation of teachers.^ 



A. MULTIPLE CHOICE DESCRIPTIVE 

Your opinion of the amount of homework for the course is: 

1. Superior, proper in amount and emphasis. 

2. Good, generally supplements work. 

3. Fair. 

4. Somewhat inadequate in value and proportion. 

5. Poor in most respects. 



B. GRAPHIC (SPECIFIC) 

Range of Interest and Culture 

10 9876543210 



Instructor has very 
broad interests and. 
culture ; frequently 
relates course to 
other fields and to 
present-day problems. 



Instructor has 
fair breadth of 
interests and 
culture; occas- 
ionally relates 
subject to other 
fields and to 
present-day 
problems. 



Instructor is narrow 
in his interests and 
culture; seldom 
relates subject to 
other fields or to 
present-day problems. 



C. GRAPHIC (VAGUE) 

Does he appear sensitive to the students’ feelings and problems? 



Unaware 



Respons ive 



L. Kent, "Student Evaluation of Teaching," The Educational Record , 
Summer 1966, pp. 384-385. 
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D. FIVE POINT SCALE 



Circle the number most appropriate (5-superior, 1-poor) 
Clarity of Speech ... 5 4 3 2 1 NA 



E. PROJECTIVE FORM 

Complete the sentences to express your real feeling: 

1. This course . . . 

2. What I liked . . . 

3. I feel that the instructor . . . 

4. If I were teaching this course I would . . . 

5. The class discussions . . . 



F. EXPLANATORY FORM 

The student should receive 50 minutes of instruction during each class 
period. An appropriate story, personal experience, or joke may illus- 
trate or emphasize a point while irrelevant stories and discussion 
unrelated to the subject waste time. How does the effective use of 
the period in this subject rank? 

The most striking impression obtained from reviewing the literature 

on evaluating teacher effectiveness is captured by Mitzel: 

'^Although research has been in progress to this end for more than a half- 
century, the effort has not yielded meaningful, measurable criteria 
around which the majority of the nation’s educators can rally. No 
standards exist which are commonly agreed upon as the criteria of 
teacher effectiveness . 



^ H. E. Mitzel, "Teacher Effectiveness," Encyclopedia of Educational 
Research (New York: The Macmillan Co., 1960) p. 148. 
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III. CRITICAL INCIDENT METHOD 






For the purpose of developing a systematic approach toward resolving 
the determinants of effective classroom instruction we selected the method 
of critical incidents coupled with content analysis. The rationale for 
favoring this method was (1) an appreciation for the comparative objectivity 
of this approach and (2) the expectation of avoiding the major pitfalls 
of biased inferences that we suspected of some studies of teacher evaluation. 

Two basic approaches to content analysis are discussed by Herzberg 

o 

et al..^ The first approach is the a priori approach basing analysis 
upon a previously defined and outlined schematic system. An example of 
this approach would be the analysis of a body of material by sorting out 
factual from evaluative material with all of the material obtained falling 
into its appropriate predefined category. 

The second content analysis approach is an a posteriori method where 
the categories of analysis are extracted from the material itself. Our 
study adopted this latter approach to better enable a division into cate- 
gories that was more meaningful in terms of the empirical material 
gathered. 



o 

F. Herzberg, B. Mausner, and others, The Motivation to Work (New York: 
John Wiley and Sons, 1967), p. 37. 
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IV. PROCEDURE 



From free written responses to a survey questionnaire we sought to 
construct categories of instructor characteristics which could express 
student perceptions of effectiveness and ineffectiveness. 

A. THE SURVEY 

Students at the Naval Postgraduate School were asked to respond to 
two questions. The first asked them to think of a time when they were 
especially satisfied with an instructor and to describe what led them to 
feel that way. The second question was identical, except that it concerned 
dissatisfaction (see Appendix A). Responses were received from 180 students 
from a distribution of 1496 questionnaires. 

B. ANALYSIS 

The second step was to sort and categorize the responses, through con- 
tent analysis, to determine mutually exclusive and nearly exhaustive 
factors leading to student satisfaction or dissatisfaction with instructors. 
This development of content categories became the central and most elusive 
aspect of the effort. On one hand, there was the danger of interpreting 
or superimposing preconceived ideas into the responses so that the structure 
and content of the categories could be made coherent. On the other hand, 
there was difficulty in classifying student perceptions of instructor 
behavior, attitudes, or efforts into categories which were really different 
from one another. 

1. Isolating Response Thoughts 

Before a response could be categorized, or grouped with others 
which were similar, a conceptual scheme was required to deal with the 
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language and processes of distinguishing the thought expressed. It became 
useful to consider an opinion as an expression of a comparison between a 
situation as it was perceived and an idea about how it ought to have been. 
Thus the questionnaire prompted the sentiment of satisfaction or dissatis- 
faction and asked for the **situation-ought** comparison. This conceptual- 
ization became useful because it required that the three mutually dependent 
elements of opinion (1. perceived situation, 2. idea of ought, and, 

3. sentiment) be accounted for. We hypothesized that those situations 
which most frequently conformed to, or deviated from, an idea about what 
the situation should have been would be most frequently mentioned by the 
respondents. Similarly, this scheme suggests that a frequency tabulation 
of instructor effectiveness determinants will yield an order-of-importance 
listing \v7hich has significance only within the context of the situations 
in which the sentiments were evoked. The mutual dependence of the particu- 
lar situation and the idea of how it ought to be affects the relative fre- 
quency of response about any one factor. Each response is considered to 
describe a situation in conformity with, or in variance from, an idea 
about how it ought to be. Our data and analysis then, insofar as they 
reflect a priority listing of effectiveness determinants, apply to this 
particular sample. If the situations in the Naval Postgraduate School 
classrooms, and/or student ideas of what should be were different, the 
relative order-of-importance listing of effectiveness determinants might 
also differ. 

2. Inference 

While clarification of the nature of opinion and its component 
elements helped conceptualize the task and acted as a brake to the tendency 
to "read into" the responses, a further problem was introduced by the 
various levels of inferences which were in fact contained in the responses. 
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On one hand, the students inferred or attributed attitudes, knowledge levels, 
and outside class effort to instructors. On the other hand, they sometimes 
inferred the reasons why the described attitudes, perceived knowledge 
levels, or attributed efforts were important to them. Inferences about 
the instructor were dealt with directly and incorporated into the cate- 
gorization scheme. Inference of the second sort, concerned with the 
reasons why an instructor’s attribute was important to the student, consis- 
tently associated satisfaction with student understanding. When explicitly 
stated, competence gain or skill acquired comments were easily classified. 

A more predominant theme was the implicit statement of, and concern for, 
comprehension which associated a behavioral or attitudinal characteristic 
of an instructor with understanding, (e.g., speaking clearly yielded 
satisfaction because it resulted in comprehension, or speaking in a clear 
manner yielded comprehension which resulted in satisfaction) . In the 
cases where the expressed sentiment was associated with understanding and 
some other characteristics, (e.g., speaking clearly, legible penmanship, 
organized presentation, etc.,) the primary characteristic was recorded. 

The secondary or associational response concerning "understanding" was 
noted so frequently (75%) that it was considered a basic common denominator 
in itself. 

3. Decision Tree 

Once the thought expressed in a response had been distinguished 
and the level or degree of inference present recognized, the categorization 
process required that it be explicitly identified. Repeated false 
starts led us to adopt a consistent procedure for categorization. Rough 
categories were etched out and refined until a decision tree evolved. In 
present form, the decision tree subjects each, recorded thought of a respon- 
dent to a series of binary criteria which attempt to capture the thought. 
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The categorizer was required to make a binary yes/no decision regarding the 
match of each thought with each set of criteria in the categorization pro- 
cess. If a match could not be made the thought was rejected and subse- 
quently reviewed. 

C. VALIDATION 

A set of instructions was written documenting the decision tree pro- 
cedure for categorization (see Appendix B). The writers, working inde- 
pendently and following the documented instruction, obtained a 95% agreement 
on classification. In addition, similar agreement was obtained when a 
panel composed of the writers and three faculty members employed the 
procedure. 
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V. FINDINGS 



The flowchart or decision tree procedure for classification and cate- 
gorization of student perceptions clarifies the determinants of instructor 
effectiveness as perceived by students. Sixteen categories have been 
developed from student responses. The structure of the flowchart or 
decision tree is simplified by five branchpoints which provide major 
decision criteria. 

A. RESPONDENTS’ UNDERLYING PREMISE 

Student concern for understanding the course material is the underlying 
theme throughout the responses. This concern is expressed explicitly as 
a determinant of satisfaction (how much was learned), directly as an 
attribute or category of instructor effectiveness (attitude toward student 
understanding) , and indirectly associated with an attribute of the instructor 
(I learned because he taught at my level) . 

B. BRANCHPOINTS 

Descriptive titles .or labels were assigned to each of the five branch- 
points within the decision tree. These branchpoint labels serve to call 
to mind the criteria upon which a binary yes /no decision must be made. 

These are not categories in themselves, but are rather convenient ways of 
thinking. They are illustrated in Figure 1 and include: 

Instructor Knowledge 

Instructor Attitude Toward (Specific) 

Instructor Classroom Presentation Ability 
Instructor Effort Outside of Class 
Learning Result 

1. Instructor Knowledge 

Includes student opinion concerning instructor proficiency in 



subject matter. 
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FLOWCHART / DECISION TREE OUTLINE ILLUSTRATING BRANCHPOINTS 
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2. Instructor Attitude Toward 



Includes student perceptions of instructor attitudes toward student 
understanding, the subject matter, student questions, or the student in 
general. A "yes” decision at this branchpoint is reached only if the 
attitude described is directed toward one of the above categories, e.g., 

"He was enthusiastic" is excluded; "He was enthusiastic about the subject 
matter" is included. 

3. Instructor Classroom Presentation Ability 

Includes student's perceptions of instructor's abilities associated 
with presentation of material in the classroom. Emphasis here is on 
instructor activity characteristically associated with classroom presenta- 
tion, and does not include instructor activities in preparation for class- 
room presentation, evaluation of students, or other efforts normally made 
by the instructor outside of the classroom. 

4 . Instructor Effort Outside of Class 

Included are student perceptions concerning instructor effort 
characteristically performed outside of the classroom. Additionally, all 
comments concerning instructor evaluation of students (e.g., examinations, 
grades, etc.) and course requirements are included. The "outside of class" 
or "effort" label is a convenient way of considering classification of a 
student perception and seems to be distinct from the actual presentation 
of material in the classroom — at least in the student's mind. 

5. Learning Result 

Includes only comments which specifically associate satisfaction 
or dissatisfaction with student competence or skill gain. Excluded are 
comments which associate understanding or learning with an attitude, 
ability, or effort of the instructor. 
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C. CATEGORIES 



Although each of the sixteen categories within the decision tree are 
subordinated to a branchpoint, they are meant to stand alone, outside of 
the context of the branchpoints which were used as a convenience when 
categorizing. The complete flowchart /decision tree is illustrated in 
Figure 2. Categories are discussed below in order of significance or 
frequency found in the survey. 

1. Organization of the Course 

This category encompasses the orientation of the course toward 
clear objectives. Included are the perceived clarity of course require- 
ments, studies to reach the objectives and relevance of the course to 
larger programs of study or ”real world** situations. Separation of important 
from less-important material and a balance between theory and the practical 
were important factors. In this category the direction of the course as 
a whole, in terms of how it actually proceeds, is central, e.g., **We 
always drifted from topic to topic, without any idea of where we were 
going or why.** 

2. Evaluation of Students 

Every aspect of' grade assignment is included in this category. Its 
position as second most frequently mentioned is dependent upon incorporating 
all aspects into one category. To distinguish the significance of compo- 
nent factors included, subcategories were constructed from those responses 
which indicated *'evaluation of students** as a factor in perceived effec- 
tiveness. The subcategories included: 

Content of examinations 
Fairness in grading examinations 
**Taxingness** of assignments 
Grading scheme or pressure on grades 
Promptness in returning examinations 
Actually reading assignments submitted 
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DECISION TREE / FLOWCHART 







These subcategories were then ranked in order of frequency mentioned with 

the other categories. With one exception, each subcategory ranked below 

\ 

all other factors of satisfaction, or dissatisfaction. The exception, 
"content of examinations," ranked tenth among all other categories and 
seemed to be the predominant issue concerning evaluation of students. 

The negative correlation between examination content and material covered 
in the course was frequently mentioned by students, e.g., "The content 
of his examinations did not resemble the material we had -covered in class." 

3. Instructor Attitude Toward Student Understanding 

This category reflects instructor concern, interest, desire, etc. 

I 

for student comprehension or progress. One extreme is reflected in the 
comment, "He doesn’t just cover topics, he wants us to learn the material." 

4. Instructor Ability to Create a Learning Environment and Demeanor 
This category captured the elusive qualities of enthusiasm, per- 
sonality, honesty, open-mindedness as well as abilities to inspire or 
motivate. The authors suspect that the ability to be appealing is probably 
the most difficult to address or influence, especially through student 
feedback. 

5. Prior Preparation 

Student perceptions regarding the degree of prior preparation for 
a classroom period are included in this category. 

6. Clarity of Explanation 

Comments concerning an instructor’s ability to clearly explain were 
categorized here. Vivid use of examples, clarity when presenting difficult 
material, and direct responses to questions were frequently noted responses, 
e.g., "His clarity (of presentation) made the difficult material as easy 
as possible." 
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7 . Instructor Knowledge 



This category includes depth and breadth of instructor proficiency 
in the course material. A frequent negative comment was, **He couldn’t 
work homework problems on the board.** 

8. Instructor Ability to Teach at Appropriate Level 

This category includes comments concerning instructor awareness 
of student level of comprehension, and adjustment of the pace and level 
of instruction appropriate to student ability and prior knowledge, e.g., 

**He taught over our heads and was unaware that no one understood what 
was going on.** 

9. Instructor Attitude Toward Course and Subject Matter 

Included are comments concerning instructor interest in and respect 
for the subject matter. Opposites in this category are **He was interested 
in the field** and **He said the course v/as useless.** 

10. Organization of Lecture Material 

The structure, logic, and sequence of material presented in class 
is included in this category. Summarizing, providing an overview, empha-* 
sizing the significant in a logical sequence, and **being flexible, without 
getting lost** are frequently-noted positive characteristics. 

11. Instructor Attitude Toward the Student 

This category refers to comments concerning instructor-student 
interrelationships in which the student discerns an attitude specifically 
toward him as a person (e.g., aloofness, patience, encouragement). Not 
included are either more specific instructor attitudes toward student 
understanding and questions or more general personality traits (honesty) 
or characteristics (**appealingness**) . 

12 . Student Competence Gain 

Only direct comments which associate satisfaction or dissatisfaction 
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with amount learned are included. "l was satisfied because 1 learned a 
lot** or **I was dissatisfied because I didn’t learn anything and the whole 
thing was a waste of time.** As previously noted, student concern for their 
understanding was implicit in a large majority of the responses. 

13. Instructor Attitude Toward Student Questions 

Expressions of instructor willingness to listen to and answer 
questions in addition to student perceptions of instructor concern for the 
content of student understanding are included in this category. Typically, 
**He wants students to understand concepts not just mechanics,** and **Never- 
mind-how-it-works-just-plug-in-the-numbers type attitude. ** 

14. Instructor Ability to Control Class and Time 

Comments concerning effective use of class time and class partici- 
pation are categorized here. 

15. Instructor Availability 

Comments concerning outside class availability to lend assistance 
to students are included, e.g., **He was always available for outside 
assistance. ** 

Instructor Ability to Evoke Interest 

The instructor’s ability to evoke interest is contrasted to, ”He 
reads from the text,” in this category. 

D. INTERPRETATION AND TABULATION 

Our findings suggest that the defined categories and procedural method 
of categorization can express student perceptions of instructor effective- 
ness in the vocabularies and within the frame of reference of students at 
the Naval Postgraduate School. The rankings of the categories in order 
of importance reflects a moment in time and a particular sample. It is 
our hypothesis that at a different moment of time or by using a different 
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sample the categories would stand while their order of ranking (according 
to their frequency) might be different. Table I presents the tabulated 
frequency of responses found in the January 1973 sample. In our estimation 
it would be erroneous to conclude from this tabulation that students are 
more satisfied than dissatisfied with instruction on the basis of positive 
or negative responses to the summation of particular categories. Similarly, 

Y it does not appear as if any of the categories represent satisfaction signif- 
icantly more than dissatisfaction, or vice versa. On the other hand, it 
may be that instructor emphasis on categories toward the higher end of the 
ranking could more quickly and effectively lead to improved effectiveness 
as perceived by students. 

Another perspective of the ranking order of categories can be gained 
if they are considered in light of either prescriptions or proscriptions 
for action. In this case two observations are suggested, 

1, Evaluation of Students 

If student evaluation or **grades** play such a dominant part in 
student perception, one questions what it is, in particular, that plays 
that vital part. Additionally, one questions how those factors related 
to student evaluation compare with the other categories. These findings 
indicate, as previously noted, that content of examinations is the central 
issue. Rearrangement of the frequency tabulation to account for the sub- 
categories or influences detected within student evaluation yields the 
rankings in Table 2, Evaluation of students in Table 2 is broken into 
six subcategories and ranked with the other fifteen categories. Exami- 
nation content then ranks tenth, while other facets of student evaluation 
are found in positions 17 through 21. 



25 



I 




ORDER-OF-IMPORTANCE CATEGORY LISTING AND FREQUENCY TABULATION 



•TABLE I 

Rank Category Total // 



Order 


Description 


Mentioned 


1 


Course Organization 


153 


2 


Evaluation of Student 


139 


3 


Attitude Toward 
Student Understanding 


112 


4 


Ability to Create 
Learning Environment 


111 


5 


Instructor Preparation 


106 


6 


Ability to Explain 


100 


7 


Instructor Knowledge 


97 


8 


Ability to Teach at 
Appropriate Level 


72 


9 


Instructor Attitude 
Toward Course 


68 


10 


Lecture Organization 


62 


11 


Attitude Toward Student 


58 


12 


Attitude Toward' 
Questions 


49 


13 


Student Learning Result 


35 


14 


Instructor Control 


34 


15 


Instructor Availability 


30 


16 


Ability to Evoke Interest 29 




TOTALS 


1255 



# of Associations # of Associations 
with Satisfaction with Dissatisfaction 



77 




76 


60 




79 


63 




49 


60 




51 


46 




60 


60 




40 


52 




45 


41 




31 


40 




28 


28 




34 


34 




24 


24 




25 


19 




16 


12 




22 


23 




7 


14 




15 


653 




602 
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2, Distinctions 



The development of the categories was an attempt to express 
determinants of instructor effectiveness as perceived by students. The 
effort has sought to make distinctions between the categories clear, 
distinct and mutually exclusive. The method has not been able to satis- 
factorily address the degree to which the categories are statistically 
different. The constructs of language and multiple phrases which can 
express the same thought may have led us to construct categories which 
seem to be different, but which may not be. The overwhelming emphasis on 
"organization** is the predominant instance. 

Comments concerning organization seemed to refer to either the 
course in general or to the lecture presentation. Organization of lectures 
was sometimes attributed to preparation, in which case it was categorized 
separately. Classifying course organization, organization of lectures, 
and preparation into one category would have been, in our minds, an 
inference. Although instructors themselves may equate the prior preparation 
of a lecture with an organized presentation, the categories have not been 
constructed as if the same cause-effect correlation is made by the student. 
The language of our branchpoints (instructor effort and instructor ability) 
kept this distinction. Thus, comments to be classified under preparation 
were recognized by the word *'prepare** whereas comments categorized under 
lecture organization referred to either the results of that preparation 
before (**His lectures were organized**) or after (**His lectures were pre- 
sented in an organized fashion**) presentation. If these three categories 
are considered one, comments concerning organization account for 25% (312) 
of the responses. 
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' E. COMPARISON WITH A SIMILAR STUDY 

As this study drew to a close, a study by W. W. Ronan reporting a 

9 

similar critical incident study became available. In comparison, the 
results of that study delineated seven subdivisions or types of instructor 
behavior (roughly analogous to our six branchpoints) and ninety-seven 
specific behaviors or actions associated with satisfaction or dissatis- 
faction (roughly analogous to our sixteen categories). The essential 
difference between the results reported herein and those found in the 
Ronan study is our concentration on student perceptions versus an 
exclusive concentration on specific instructor actions. 



^ U. S. Department of H. E. W. /Office of Education, PREP Report No. 34, 
Evaluating College Classroom Teaching Effectiveness , by W. W. Ronan. 
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ORDER-OF-IMPORTANCE CATEGORY LISTING AND FREQUENCY TABULATION 
SUBCATEGORIZING FACTORS IN STUDENT EVALUATION 

TABLE 2 



Rank 

Order 


Category Total // 

Description Mentioned 


// of 
with 


Associations 

Satisfaction 


1 


Course Organization 


153 




77 


2 


Attitude Toward 
Student Understanding 


112 




63 


3 


Ability to Create 
Learning Environment 


111 




60 


4 


Instructor Preparation 


106 




46 


5 


Ability to Explain 


100 




60 


6 


Instructor Knowledge 


97 




52 


7 


Ability to Teach at 
Appropriate Level 


72 




41 


8 


Instructor Attitude 
Toward Course 


68 




40 


9 


Lecture Organization 


62 




28 


10 


Examination Content 


61 




22 


11 


Attitude Toward Student 


58 




34 


12 


Attitude Toward Questions 


49 




24 


13 


Student Learning Result 


35 




19 


14 


.. Instructor Control 


34 




12 


15 


Instructor Availability 


30 




23 


16 


Ability to Evoke Interest 


29 




14 


17 


Fairness in Grading 
Examinations 


28 




12 


18 


Taxingness of Course 


21 




10 


19 


Grading Scheme 


17 




10 


20 


Promptness Returning 
Examinations 


11 




5 


21 


Attention to Submitted Work 2 




1 



1255 653 

29 



Associations 

Dissatisfaction 

76 

49 

51 

60 

40 

45 

31 

28 

34 

39 

24 

25 
16 
22 

7 

15 

16 

11 

7 

6 

1 

602 



TOTALS 



VI. APPLICATIONS 



Two applications of the findings to enhance feedback to instructors 
are suggested. 

A. GENERAL FEEDBACK 

The development of a document which could make available the comments 
of students recorded during the survey in the format of the developed cate- 
gories might prove of some value to instructors seeking to enhance their 
effectiveness. To this end a general description of our efforts and ver- 
batim comments classified according to developed categories has been com- 
posed and is included as Appendix C. In addition, this appendix provides 
documentation for the classification procedure described earlier. 

B. SPECIFIC ROUTINE FEEDBACK 

A vehicle and a procedure which could be routinely employed to gather 
student perceptions or opinions of instructor effectiveness could be 
based on the categories distilled from this effort. Development of 
suitable scales awaits future efforts. 
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APPENDIX A 



SURVEY QUESTIONNAIRE 

The instructor can make or break the course. Student opinion too often 
travels via the grapevine of acquaintances instead of back to the instructor 
or to those making administrative decisions. If student evaluation is to 
be significant, the determinants of student evaluation must be clarified. 
This questionnaire seeks to clarify what should be evaluated or measured 
about an instructor’s performance from the perspective of the NPS student. 

Your comments on page two will assist in this endeavor. Identification 
of instructors by name is specifically NOT solicited. Please leave the 
completed questionnaire in the ’’instructor questionnaire” box in the 
Student Hail Center. 

This questionnaire is distributed lAW NPS INST. 1520.13 by students 
fulfilling the requirements for MN 0810, Thesis Research, under the 
direction of Professor R. S. Elster. 

Thank you for your assistance. 
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Curriculum Qtrs Completed at NFS 

QPR Overall Yrs Military Service 



Think of a time when you were particularly satisfied with an instructor. 
What led up to this feeling? 



Think of a time when you were not satisfied with an instructor. IThat 
led up to this feeling? 
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APPENDIX B 



INSTRUCTIONS FOR CATEGORIZING STUDENT RESPONSES TO 
CRITICAL INCIDENT QUESTIONNAIRES CONCERNING INSTRUCTION EFFECTIVENESS 

I. INTRODUCTION 

/ 

Questionnaires soliciting student opinions concerning the factors of 
instructor effectiveness at the Naval Postgraduate School require processing. 
Each returned questionnaire contains free-written responses which indicate 
factors important to the respondent. The task is to identify like factors 
and group them into categories. 

The categorization process involves two steps: first, distinguishing 

the thoughts within the response, and second, categorizing the distinguished 
thoughts into categories containing similar thoughts. Certain concepts 
and procedures assist in this process. 

II. ISOLATING THE THOUGHT 

Conceptual clarification of the nature of the student responses can 
be helpful. Students have recorded their opinions regarding positive and 
negative aspects of instructor effectiveness. These opinions consist of 
three elements, some or all of which may be present in any given response. 
When seeking an opinion, a response indicating like or dislike, in itself, 
is considered to be a comparison between the situation called to mind and 
the sentiment evoked. If the reason for the sentiment is explained, that 
explanation or elaboration makes a comparison between the situation as it 
is perceived and an idea about how it should have been. To a large extent, 
the responses, and hence the categories into which they are classified. 
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represent variances from, or conformity with, ideas about ’’how tilings 
should be,” 

Each categorizer must consciously avoid reading his own thoughts or 
past experiences into responses. The responses must be taken at their 
face value and not interpreted to represent thoughts consistent with the 
experience or bias of the categorizer. This is more easily said than done. 
It is the individual thoughts that must be categorized, not the general 
gist of the response. 

III. CATEGORIZATION OF THE THOUGHT 

The categorization procedure involves processing each distinguished 
thought (which may be a phrase, a sentence, or a paragraph) through a 
flowchart which has five branchpoints and sixteen categories. At each 
branchpoint and each category a binary yes/no decision is required to 
determine the next path or appropriate action. Each decision is based 
on criteria within the decision box. If the criteria within the decision 
box ’’capture” the thought being processed, follow the ”yes” path. If not, 
follow the ”no” path. Each thought is allowed one pass through the flow- 
chart; if uncaptured it ends in a reject box and is logged accordingly. 
Branchpoints have the one-digit numbers one through five. Categories 
have two digit numbers. Each category is subordinate to one branchpoint, 
although two of the five branchpoints have only one category (1 has 11, 
and 5 has 51) . 

Each branchpoint and category is described by a set of criteria upon 
which the yes /no decision is based. In addition to the criteria and a 
number, each also has a word or name which serves the function of calling 
the criteria to mind. 
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The branchpoints are not categories; they merely assist in making 
the correct decision based on the criteria they contain. Again, the 
names serve to call the criteria to mind. The branchpoints are numbered 
and labeled as: 

1 Instructor Knowledge 

2 Instructor Attitude Toward 

3 Instructor Classroom Presentation Ability 

4 Instructor Effort Outside Class 

5 Student Learning Result 

A condensed version of the flowchart is presented in Figure A-1. 

Each thought is processed through the flowchart until it is either 
captured by the criteria of a category or is rejected. Once captured, 
the corresponding number of the category is logged on a three-by-five 
inch card sequentially numbered to correspond with each questionnaire. 
Figures A-2 and A-3 illustrate the format. The category numbers of 
thoughts associated with satisfaction (the top half of the questionnaire) 
are logged on the left half of the card; those associated with dissatis- 
faction (the bottom half), on the right. If a thought is rejected, the 
response is logged verbatim on the reverse side of the card. 

Figure A-4 displays the entire flowchart/decision tree and includes 
category labels. The addendum includes the criteria which these labels 
represent and examples of thoughts included and excluded in each category. 
As a procedural method, the addendum should be disassembled and organized 
into a single chart. Figure A-4 will then serve as a condensed mapping 
reference. Prior familiarity with the criteria is essential and organi- 
zation into a single diagram facilitates categorization. 

To summarize, the categorizer has before him sequentially numbered 
survey questionnaires, correspondingly numbered cards, a schematic repre- 
sentation of the flowchart, and Figure A-4. Thoughts expressed in the 
questionnaires are processed through the flowchart. The number of the 
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FLOWCHART / DECISION TREE OUTLINE ILLUSTRATING BRANCHPOINTS 



START 



KNOWLEDGE 



ATTITUDE 




ABILITY: CLASS _ 


. EFFORT 




LEARNING 


TOWARD 




PRESENTATION 






RESULT 
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FIGURE A-1 
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Category numbers 


Category numbers 


Sequential 


corresponding to 


corresponding to 


number 


the thoughts ex- 


the thoughts ex- 




pressed in the 


pressed in the 




top half of this 


lower half of 




questionnaire . 


this questionnaire . 





Figure A-2 



21 


31 


087 


33 


46 




11 


43 




24 






51 







Figure A-3 
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DECISION TREE / FLOWCHART 







category capturing each thought is logged on the appropriate half of the 
card. If a thought is rejected, it is recorded on the back side of the 
card. 



IV. CONCEPTUAL AIDS 

Initial difficulty recording comments concerning “organization” and 
"student understanding” can be anticipated. Distinctions between comments 
addressing either of these subjects follow specific ruleg within the flow- 
chart. 

A. STUDENT UNDERSTANDING 

Underlying many of the responses is the implicit expression of the 
student ^s idea that he ought to learn and that the instructor ought to 
teach in a manner such that learning results. These kinds of thoughts 
are handled in two ways. If the thought directly, and without qualification 
or association with an instructor attribute, expresses satisfaction or 
dissatisfaction on the basis of student competence gain, it is categorized 
under "Learning Result” (51). For example, "I was satisfied because I 
learned the subject material" or "The course was a complete waste of time 
and I didn’t learn a thing." On the other hand, if understanding is 
addressed in the context of, or is associated with, an instructor attribute, 
the thought is logged according to the attribute with which it is associated. 
For example, "He was concerned with my understanding" (21) or "His explana- 
tions were clear and I could understand them” (32), 

B. ORGANIZATION 

A second predominant theme in the responses concerns organization. 

The decision tree distinguishes between three closely related concepts. 
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The first distinction rests between the organization of the course (44) 
and the organization of the lecture (34). While the distinction seems 
clear enough, course organization is a broader concept and includes 
general comments about organization (e.g., **He was disorganized."). More 
specific comments may refer to either the course as a whole or the class- 
room presentation of the material. A second distinction rests between 
preparation (41) and lecture organization (34). The category including 
preparation is almost always recognized by the word "prepare." Lecture 
organization, on the other hand, denotes both the results of preparation 
("His lectures were organized") and the results of presentation ("He 
presented the material in an organized manner."). Although it seems 
logical to infer that preparation results in an organized lecture presenta- 
tion, it is an inference nevertheless. To the greatest extent possible 
the categorization procedure seeks to avoid inferences on the part of the 
categorizer . 



V. LOGGING RULES 



When recording the numbers of categorizers which have captured thoughts, 
any given category number may be recorded only once for the positive and 
once for' the negative expression in any one response. Thus, if the same 
thought is expressed in several sentences it is recorded only once vice 
once for each sentence. In those instances when the respondent has 
indicated "the converse of the above" on the bottom half of the question- 
naire, each category number recorded on the left side of the card is also 
logged on the right. Several practice runs through the procedure, adhering 
to the yes /no decision at each point in the flowchart, are recommended. 
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ADDENDUM I TO APPENDIX B 



FLOWCHART BRANCHPOINT AND CATEGORY DECISION 

CRITERIA 
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PAGE 1 



START HERE 



INSTRUCTION 
Go to page 2 
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Entry from page 1 



PAGE 2 



Branchpoint label: KNOWLEDGE (1) 

Subordinate Categories: Instructor Knowledge (11) 

Criteria: Instructor knowledge of the subject he is teaching 

Knowledge in his field 

Includes depth and breadth of knowledge 
Demonstrated proficiency in subject matter 

Examples: ”He was knowledgeable in his field," 

"He couldn*t work homework problems on the board," 

"He V7as unable to relate the subject matter to the real world." 

This category does not include statements such as 

"He did not relate the subject matter to the real world." or 

"He did not work homework problems on the board." 



INSTRUCTION/DECISION 
"YES" go to page 3 
^*N0" go to page 4 
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Entry from page 2 



PAGE 3 



Category: INSTRUCTOR KNOWLEDGE (11) 

Superior branchpoint: KNOWLEDGE (1) 

Criteria: The criteria for this category are the same as the criteria for 

the branchpoint. Has the thought been captured? 



INSTRUCTION/DECISION 
"YES" log category'TTl 
"NO" reject 
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Entry from page 2 



PAGE 4 



Branchpoint label: ATTITUDE TOWARD (2) 

Subordinate categories: Student understanding/teaching (21) 

Subject matter/field (22) 

Student Questions (23) 

Students (24) 

Criteria: Attitudes, but must be directed toward one of the subordinate 

categories . 

Words which help identify attitudes include enthusiasm for, 
interest in, concern for, willingness to, involvement in, 
desire to, and wants. 

Examples: ”He was encouraging” YES 

”He was enthusiastic” NO 

”He was enthusiastic about the subject” YES 
”He really wants to teach” YES 
”He was aware of . . NO 

”He adjusted the pace” NO 

”He adjusted the pace because he really wanted us to understand” 
”He cares about what we learn” YES 



INSTRUCTION/DECISION 
”YES” go to page 5 
”N0” go to page 9 



YES 
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Entry from page A 



PAGE 5 



Category: STUDENT UNDERSTANDING (21) 

Superior branchpoint: Attitude toward (2) 

Criteria: Attitude toward student progress 

Attitude toward teaching 

Attitude toward student comprehension of material 
Examples: **He doesn’t just cover topics, he wants us to learn the material.” 



INSTRUCTIQN/DECISIQN 
”YES” log category 21 
”N0” go to page 6 
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Entry from page 5 



PAGE 6 



Category: SUBJECT MATTER/FIELD (22) 

Superior branchpoint: Attitude toward (2) 

Criteria: Interest in the field 

Respect for the field 
Attitude toward course material 

Examples: "He said the course was useless." 

"He was interested in the field." 



INSTRUCTION/DECISION 
"YES" log category 22 
"no" go to page 7 
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Entry from page 6 



PAGE 7 



Category: STUDENT QUESTIONS (23) 

Superior branchpoint: Attitude toward (2) 

Criteria: Willingness to listen to and answer student questions 

Concern for the content of student understanding 

Examples: **He is offended by questions." 

"He wants students to understand concepts not just mechanics." 
"Never-mind-how-it-works-just-plug-in-the-numbers type attitude. " 

Not included in this category are comments such as 
"He teaches concepts not just mechanics." 



INSTRUCTION/DECISION 
"YES" log category 23 
”N0" go to page 8 
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Entry from page 7 



PAGE 8 



Category: STUDENT (24) 

Superior branchpoint: Attitude toward (2) 



Criteria: His manner in relation to students, but not his manner in general. 

Implies a specific trait in relation to students. 



Examples : 



**He is aloof." 

"He is encouraging." 

"He is patient." 

"He has a superior attitude." 



But not, 

"He has a good personality." 
"He has colorful speech." 

"He is a good showman." 

"He is enthusiastic." 

"He is honest." 

"He was appealing." 



Comments such as "He was inspiring" do not belong in this category. 
"He made me want to learn" is thought of as an ability vice an 
attitude of the instructor and is subordinated to branchpoint 3. 



INSTRUCTIQN/DECISIQN 
^*YES" log category 24 
^^NO" reject 
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Entry from page 4 



PAGE 9 



Branchpoint label: CLASSROOM PRESENTATION ABILITY (3) 

Subordinate Categories: To teach at appropriate level (31) 

To explain (32) 

To evoke interest (33) 

To organize lecture (34) 

To control time/participation (35) 

To create a learning environment and demeanor (36) 

Criteria: Characteristic of material presentation in the classroom during 

a class period 
Abilities or attributes 

Examples: ”He adjusted the pace.** 

**He read from the text.** 

**He was enthusiastic.** 

**His board work was neat and legible.** 

**He is aware of the degree of class understanding.** 

**He puts material across.** 

**He made me want to learn.** 

**He was narrowminded.** 

**He speaks well.** 

**He is colorful.** 

**His lectures were presented in an organized manner.** 

**His lectures were organized.** 

Not included in this category are comments reflecting preparing 
for lectures, organizing the course, and evaluating the student. 
Although the student may observe these in the classroom they 
are thought of as characteristic activities of the instructor 
outside of class. Examples are **He was organized.** (see 44) 
and **He was prepared.** (see 41). 



INSTRUCTIQN/DECISIQN 
**YES** go to page 10 
**N0^* go to page 16 
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Entry from page 9 



PAGE 10 

Category: TO TEACH AT APPROPRIATE LEVEL (31) 

Superior branchpoint: Classroom presentation ability (3) 

Criteria: Teach at the level of student knowledge 
Teach at the level of student ability 
To be aware of student comprehension 

Examples: **He used layman’s terms.” 

”He could adjust the pace.” 

”Just right as far as challenge goes.” 



INSTRUCTION/DECISION 
”YES” log category 31 
”N0” go to page 11 
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Entry from page 10 



PAGE 11 

Category: TO EXPLAIN (32) 

Superior branchpoint: Classroom presentation ability (3) 

Criteria: Clarity of explanation 

Good imparter of knowledge 
Ability to answer questions 
To use examples to clarify 
To make material understandable 



INSTRUCTION/DECISION 
"YES" log category 32 
"NO" go to page 12 
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Entry from page 11 



PAGE 12 

Category: TO BE INTERESTING (33) 

Superior branchpoint: Classroom presentation ability (3) 

Criteria: To evoke interest 

To make material interesting 

Examples: The most frequent negative comment in this category is 

"He reads from the text." 



INSTRUCTION/DECISION 
"YES" log category 33 
"NO" go to page 13 
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Entry from page- 12 



Category : 

Superior 

Criteria: 

Examples : 



PAGE 13 



TO ORGANIZE LECTURE MATERIAL (34) 

TO PRESENT MATERIAL IN AN ORGANIZED MANNER 

branchpoint: Classroom presentation ability (3) 

Structure, sequence, and **logicalness’* of material presented 
during a lecture 

Pointing to significant points in a lecture 
Summarizing, overviewing, emphasizing 
Being flexible, without getting lost 

Does not include organization or composition of the course 
Not to be confused with preparedness, which may or may not 
result in an organized presentation 

Not to be confused x^rith more general comments about organization 
such as ”He was organized.** 



INSTRUCTION/DECISIQN 
^*YES*' log category 34 
**N0** go to page 14 
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Entry from page 13 



PAGE 14 

Category: TO CONTROL TIME/PARTICIPATION (35) 

Superior branchpoint: Classroom presentation ability (3) 

Criteria: Effective use of class participation 
Effective use of class time 

Control of direction and content of classroom instruction 



INSTRUCTION/DECISION 
"yes" log category 35 
"NO" go to page 15 



Entry from page 14 



PAGE 15 

Category: TO CREATE LEARNING ENVIRONMENT AND DEMEANOR (36) 

Superior branchpoint: Classroom presentation ability (3) 

Criteria: To inspire, motivate 

To exemplify appealing attributes, traits, characteristics 
To present a learning demeanor 
To create a learning environment 

Examples : **Shovnnanship" 

"Personality” 

"Enthusiasm" 

"Honesty" 

"Blackboard techniques" 

"He made me want to learn." 

"He talks too much." 

"He always tries to prove how smart he is." 

"Narrowminded" 

"Rapport" 



INSTRUCTION/DECISION 
"YES" log category 36 
"NO" reject 
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Entry from page 9 



PAGE 16 



Branchpoint label: EFFORT OUTSIDE CLASS (4) 

Subordinate categories: To prepare (41) 

To be available (42) 

To evaluate students (43) 

To organize course (44) 

Criteria: Instructor responsibilities characteristically performed outside 

of the classroom 

All references to grades, examinations, evaluation of students, 
and **taxingness** of work 

Relevance and clarity of course objectives and requirements 
Emphasis and relevance of course material 
General remarks concerning organization 
Preparation of material for classroom presentation 



INSTRUCTIQN/DECISION 
**YES” go to page 17 
”N0” go to page 21 
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Entry from page" 16 



Category: TO PREPARE (41) 

Superior branchpoint: Effort outside class (4) 
Criteria: Instructor preparedness for lectures 
Examples : Usually recognized by word "prepare" 



PAGE 17 



INSTRUCTION/DECISION 
"yes" log category 41 
"NO" go to page 18 
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Entry from page 17 



PAGE 18 



Category: TO BE AVAILABLE (42) 

Superior branchpoint: Effort outside class (4) 

Criteria: Offering outside assistance 

Being available for help 
Readiness to assist 



INSTRUCTION/DECISION 
*'YES" log category 42 
**N0” go to page 19 



N 
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Entry from page 18 



PAGE 19 



Category: TO EVALUATE STUDENTS (43) 

Superior branchpoint: Effort outside class (4) 

Criteria: Content and composition of examinations 

Fairness grading examinations 
"Taxingness” of work 

Grading scheme, including emphasis or pressure on grades 
Reading submitted assignments 
Promptness in returning examinations 



INSTRUCTIQN/DECI S ION 
^’YES" log category 43 
"NO" go to page 20 
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Entry from page 19 



PAGE 20 



Category: TO ORGANIZE COURSE (44) 

Superior branchpoint: Effort outside class (4) 

Criteria: Cohesiveness of course 

Clarity of course objectives and requirements 

Separation of important material from less important 

Orientation of course: Theory versus Practical 

Relevance to real world 

Relevance of assignments to objectives 

General comments concerning organization 

Tendency of course material to be a unit 



INSTRUCTION/DECISION 
”YES*‘ log category 44 
”N0” reject 
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Entry from page 16 



PAGE 21 



Branchpoint label: LEARNING RESULT (5) 

Subordinate category: Competence gain (51) 

Criteria: Explicit statement of satisfaction as a result of the amount 

learned, skill acquired, competence gained, a learning experi- 
ence, or understanding obtained. 

Does not include comments which associate resultant understanding 
with another instructor attribute. 



INSTRUCTION/DECISION 
**YES" go to page 22 
’*N0" reject 
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Entry from page 21 



PAGE 22 



Category: COMPETENCE GAIN (51) 

Superior branchpoint: Learning result (5) 

Criteria: The criteria for this category are the same as those for the 

branchpoint. Has the thought been captured? 



INSTRUCTION/DECISION 
"YES" log category 51 
"no" reject 
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APPENDIX C 



STUDENT PERCEPTIONS. OF INSTRUCTOR EFFECTIVENESS 

Different openings for this discussion of students* perception of 
teachers have been pondered, like ** . . • what you’ve always wanted to 
know about students* perceptions but were afraid to ask • . •** or, ** . . • 
just one hour toward a more tantalizing teaching image • . .** or, perhaps, 

** • . • how to be one up on your colleagues on student opinion of teaching 
questionnaires at NPS ..•.** However, any of these statements might be 
considered rather cheeky. It seems better to dispense with the statements 
and the discussion, and bore in "on the facts;" here they are. 

In an attempt to clarify the factors of instructor effectiveness as 
perceived by students a survey was conducted in January 1973. All Naval 
Postgraduate School students were asked to respond to two questions: 

1. Think of a time when you were particularly satisfied with an instruc- 
tor. What led up to this feeling? 

2. Think of a time when you were not satisfied with an instructor. What 
led up to this feeling? 

Student responses were categorized according to content in an effort to 
group similar answers. Once these categories were developed, it was 
envisioned that a vehicle could be developed in which routine feedback to 
instructors could be provided. Secondly, a more general feedback could be 
obtained if the kinds of responses received to the questionnaire in January 
were made available. These pages seek to make that general kind of feed- 
back available. 

The effort to group responses such that a large number of student 
perceptions could be coherently expressed resulted in sixteen categories. 
Although the development of these categories was the major effort associated 
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with the survey, the verbatim student responses to the questionnaire are 
provided here to illustrate the kind of comments received* Several pre- 
dominant themes seem to underlie these responses. 

A. Either implicitly or explicitly stated, student understanding of 
course material, or facilitation of student comprehension seemed to be the 
bases upon which satisfaction or dissatisfaction were expressed. Particular 
attitudes or actions on the part of the instructor were most frequently 
singled out because of the contribution they were felt to make to student 
understanding. 

B. There seemed to be an overwhelming student emphasis on organization of 
the course and classroom presentation. This emphasis was manifested in 
comments concerning organization of the course, organization of the class- 
room presentation, and prior preparation, which, for purposes of this study, 
were categorized separately. Student experience in a military environment 
may play some part in this emphasis. 

C. Each of the categories was associated about as frequently with student 
satisfaction as with student dissatisfaction. 

D. As might be expected, comments associating satisfaction or dissatisfaction 
with the evaluation of students by instructors were frequently noted. Further 
subdivision of this category of responses indicated that a main issue in 

the students’ minds was the content of examinations, i.e., appropriateness 
of material, difficulty, length, etc.. 

E. The categories and verbatim responses that follow are presented in 
order of their frequency of mention. 
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!• Course organization 

2, Instructor evaluation of student 

3. Instructor attitude toward student understanding 

4* Ability to create learning environment /demeanor 

5. Instruction preparation 

6. Clarity of explanation 

7. Instructor knowledge 

8. Ability to teach at appropriate level 

9. Instructor attitude toward course 

10. Lecture organization 

11. Instructor attitude toward student 

12. Instructor attitude toward questions 

13. Student competence gain 

14. Instructor control of class time 

15. Instructor availability 

16. Ability to evoke interest 

While the categories might be expected to reflect the factors students 
perceive as being associated with effectiveness or ineffectiveness of 
teaching in any sample, one would not necessarily expect the ranking to 
remain the same in another sample, say at another school, or, at another 
time. For purposes of illustration, the context of student remarks has 
often been included below even thought the surrounding context might have 
otherwise fit into another category during the classification procedure. 
Verbatim student comments follow. 
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ORGANIZATION OF THE COURSE 



Courses where the student must grope for what the course is to cover 
because the instructor has failed to structure an outline to conform 
with the catalogue; i.e., the instructor seems to lecture on whatever 
strikes his fancy, no course outline prepared, avoids questions related 
to course content. 

Provided the student with an overview of what he was trying to teach, 
its place in the overall discipline, the importance or reasons for 
studying it, and its applications. 

He clearly defined the course and what was expected in the way of home- 
work, tests, papers, etc., from the outset. 

An outline of the course was presented, at the beginning, with things 
that resembled course objectives. The material seemed pertinent to 
our total field of study. Reasonable assignments were given that 
reinforced and complimented classroom work. 

Disorganization — skipping around from section to section in a text during 
the lecture and then from lecture to lecture, without stating where 
lecture material is coming from and without providing daily homework 
assignments to either supplement the previous day’s lecture or prepare 
for the next one. 

The material was taught from a practical point of view rather than a 
series of mental gymnastic exercises. The Prof had some idea about 
how the subject might be applicable to the Navy. 

He knew what he wanted to teach and the procedures by which he wanted 
to achieve his goal. He indicated he knew the good and bad points about 
the text and had additional material for the parts where it was needed. 



EVALUATION OF STUDENTS 



The student was well informed from the beginning exactly what was expected 
from him — and the instructor followed through by evaluating the student 
on those expectations. 

His exams reflected the material presented. His grading was fair and 
impartial. 

Tests, projects which gave an objective evaluation of my performance. 

This does not necessarily mean tests which were graded favorably. 

He was demanding in time and effort and his exams required a comprehensive 
knowledge. 
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Homework assignments were meaningful additions to his classroom presen- 
tation and the book, and not just busy work. 

Testing on what appears to be insignificant matters not covered by text 
or lecture in any degree of detail. 

Grading: either arbitrarily easy (i.e,, much partial credit for obviously 

incorrect work, apparently in an attempt to camouflage poor instruction,) 
or arbitrarily hard or inconsistent, and a supercurve. 

Examinations that were made to test the material as the student would 
be later using it vice emphasizing regurgitation, 

jv Made many mistakes on the board while doing problems yet expected each 
answer to be exactly correct on exams. 

By appearing “hard” all quarter he managed to completely kill interest 
in the subject matter and then award all his students with A’s and B’s 
— not indicative of levels of learning at all. 



INSTRUCTOR ATTITUDE TOWARD STUDENT UNDERSTANDING 



The instructor displayed, and held, a genuine positive interest in the 
students’ progress in obtaining an understanding and knowledge of the 
subject material which he knew would be essential to the successful 
completion of follow-on course. Through individual instruction, 
required problems, and frequent quizzes he monitored individual progress 
and was able to immediately spot any trouble spots. 




My belief that the instructor really cared whether or not the student 
understood the subject matter. He slowed down, repeated, answered the 
’’stupid” questions until the student really did understand the subject. 
He gave the impression that he was happy to spend extra time with the 
student to insure the student learned. 



His sincere interest in imparting knowledge to his students vice just 
presenting material to meet the requirements of the curriculum. His 
goal for his students was understanding not memorization. 



A desire to see the student learn and understand material that is pre- 
sented, rather than to keep moving ahead just to remain ”on schedule” 
and heck with understanding the material. 



He was interested only in ending the period and meeting requirements. 

He didn’t really care if the students learned, 

. , . just ’’fulfilling his duty” by covering the topics instead of trying 
to teach us something. 



The real key to his success was that he was sincerely interested in the 
students understanding the material and made great efforts both inside 
and outside of class to insure this goal was achieved. 
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When I was able to determine that the professor’s goal v;as to teach each 
and every individual in the class; via extra help sessions, office hours 
that were realistic, personal attitude, and receptivity. 

He realized his main function was instructing not research, writing 
papers, or writing a book. 



INSTRUCTOR ABILITY TO CREATE A LEARNING ENVIRONMENT AND DEMEANOR 



He always maintained a good humor, inserting personal examples of 
trying times he himself experienced so that the student was never 
made to feel stupid, ignorant, or thick. This rapport created a 
"learning atmosphere" highly conducive to education. 

The course was a required introductory course but completely unrelated 
to my major. My initial feelings were of the necessary evil, let’s 
get it over with. About the middle of the quarter I discovered a keen 
interest in the subject had been kindled by the instructor and I wanted 
very much to pursue the subject. This kindling v;as done by the instruc- 
tor’s use of personal experiences of about 20 years in the application 
of the discipline. He took abstract theory and made it live. 

He talked loudly enough so the whole class could hear. His board work 
was legible and clearly organized. He went slowly enough so that we 
had time to take notes and listen to the presentation. Handouts were 
used as an aid to understanding. 

The instructor's enthusiasm for the material being taught coupled with 
a genuine concern that the students were learning the material. 

Refusal to admit that he didn’t know a point and failure to provide 
the information at a subsequent class session. 

A course in which my major concentration seemed to focus on the dis- 
tracting mannerisms of the instructor. Rather than listening to what 
he said, I found myself counting the number of "ah’s" and ’’it happens 
to be’s" . . , unfortunately, these were not testable and my final 
mark was an indication of what I had concentrated on. 

His ability to take a dry, methodical subject and by giving sidelights 
as to the utility of the subject in wide areas, make the course an 
enjoyable experience. 

He was not constantly trying to prove how smart he was. 



INSTRUCTOR PREPARATION 



He was totally prepared for each lecture; i.e., examples were worked 
out in advance, homework problems were worked by the instructor prior 
to assignment (thereby avoiding lengthy exercises of questionable worth), 
student difficulties and questions were anticipated. 
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Trying to bullshit his way through class without having so much as read 
the assigned text. 

He simply didn’t spend the necessary time to prepare his lectures in a 
manner that would enable the student to grasp the lecture material pre- 
sented. 

Instructor totally prepared — pointed out problem areas before encountered. 

His continual confusion of basic principles due to lack of preparation. 

When the instructor spent 45 minutes of the classroom time working a 
homework problem and still didn’t solve it. He was unprepared. 

Instructor assigned a problem for specific class period. On the day 
assigned, the instructor was not prepared to discuss the problem. He 
evaded direct questions and kept jumping to new material. The instructor 
asked for a consensus of opinion on the part of the students what ’’the 
most popular answer” was. 

The overwhelming emphasis on presenting material vice imparting knowledge. 
I am sure all my instructors possess a thorough knowledge in their 
fields. However, the majority lack in their ability to impart that 
knowledge. 



CLARITY OF EXPLANATION 



Outstanding lecture presentation that was clear, concise, and to the 
point. Most of all, his orderly manner of presentation from point A 
to point B and then a summary of high points during the last minutes 
of class. 

Most importantly, he presented the material in a manner that enabled 
me to understand a subject I had previously found very difficult. 

His willingness to offer complete explanation of classroom material 
when he could and his open admission that he couldn’t when he couldn’t. 

He could really get difficult concepts across to the class even if 
several different types of explanations were needed. He didn’t leave 
anyone hanging. 

He was unclear, vague and often changed his position on statements. 

One never knew for sure if his class notes were correct or complete 
or even applicable. 

His ability to draw on previous knowledge of the class to illustrate 
his points. 

He never finished explaining anything that he started. His tests 
appeared to be some of the things that he started to explain but 
never finished. 
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He has a way of saying things simply, and' relating them to things I’m 
familiar with. 

He could explain homework problems clearly. He illustrated high points 
with examples other than those read for homework. 

He acted self-satisfied, smug, and spouted a lot of jargon. He explained 
things I already knew, yet he managed to make them sound hopelessly 
complicated i 



INSTRUCTOR KNOWLEDGE 



The instructor not only knew the texts well, and could. quote directly 
and accurately from them, but he had just completed several years in 
the course environment. He had an opportunity to get a good compre- 
hensive view of his subject. HE WAS CURRENT. 

The feeling that the instructor knew what he was talking about and not 
merely reading out of the text. He could expand on what was in the 
text because of experience and competency. 

Intimate and unshakable knowledge of the area he was teaching. 

It was obvious that the teacher had no current exposure of real world 
problems, just the academic approach. 

INCOMPETENCE! e.g., a professor who can't do a problem on the board 
when the majority of the class can do it because they took the effort 
to read the pertinent section in the text. 

A feeling the instructor didn't know the material any better than I did. 

The instructor lived in an academic vacuum with no idea how his course 
should or could relate to the world of a Naval Officer. 

A demonstrated inability in the area of instruction such as constant 
errors working sample problems. 

Instructor unsure of the material. He became extremely nervous when 
asked questions. His extemporaneous ramblings left me cold. 

Assigned work without having done it himself — consequently embarrassed 
when asked to do a problem and was not able to. 



INSTRUCTOR ABILITY TO TEACH AT APPROPRIATE LEVEL AND PACE 



He first established a common base of knowledge within the class. Even 
though he had to go back to a somewhat elementary level, the class was 
then with him all the way, and progress was very rapid and real. 
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Emphasis on getting the material covered vice teaching. 

The instructor did not move on to new material until he was satisfied 
everyone in the class understood. 

His ability to identify how well the class was understanding the material 
he was teaching. 

He would tell us over and over how important it was to keep moving 
whether we understood the previous material or not. 

The instructor lectured at a level consistent with his background and 
expertise, but right over the heads of many in the class. 

He felt compelled to stick to a daily routine, even if it meant covering 
an extraordinary amount of difficult material in one or two days. 

He gloried in presenting abstract mathematical theories and persisted 
even though it should have been obvious that he had lost 80 - 90% of 
the class. 

The instructors did not draw on my previous knowledge, and taught 
below my level of intelligence. 

He spoke on my level, even though the material was over my head. 

The instructor had no feeling of how well the class was progressing 
and continuously glossed over information which the class is not 
understanding . 

Did not appear to be interested or aware that many students were 
failing to grasp important concepts of the course. 



INSTRUCTOR ATTITUDE TOWARD COURSE AND SUBJECT 



He failed to exhibit any enthusiasm in the basic course he was teaching 
and just ’^showed up” for each class. 

He was quite proud of his field, and, as a result, his students did 
not feel they were learning something irrelevant in the real life 
world. The Prof consistently related his topics to Navy-oriented 
problems . 

He was interested in his subject and projected this interest to me. 

I felt the instructor wasn't interested in what he was teaching. He 
seemed to have other things on his mind besides the class. He knew 
the material but couldn't teach. 

He acted like he was enjoying himself. He was full of his subject. 
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When the professor commenced the course with an apology for having to 
teach the course and for the students having to take the course. 

He was continually downgrading the importance of the subject. 



ORGANIZATION OF LECTURE MATERIAL 



He was organized. He explained things from beginning to end in a 
logical step-by-step process so that nothing was skipped and no doubts 
remained. 

The lectures were well organized and the presentation was held to the 
subject matter and the session not allowed to ramble oT wander. 

The instructor obviously had it all together. I felt that he knew 
much more about the subject but was giving us the cream of what he 
knew in a well organized and clear fashion. (Not designed to snow 
students but to instruct them.) 

The instructor was disorganized and would often drift off the pertinent 
subject matter. So much so that with two weeks left in the course we 
had five weeks worth of important valuable material to cover. You can 
guess the result. 

Erratic, broken lectures resulting from inadequate organization. 

He presented the material in a manner such that the relevant material 
was stressed and nice to know material was not stressed so much. 

The instructor had a brilliant mind and a deep knowledge of the subject. 
But his presentations were so unorganized, his chalkboard work so jumpy 
from place to place, and his ability to orally communicate so limited 
(he would stop in mid-sentence to interject another thought and never 
return to the subject at hand) that he succeeded only in confusing 
the student. 

He continually went on unimportant tangents in lectures. 



INSTRUCTOR ATTITUDE TOWARD STUDENT 



The instructor treated me as an expert in my own field of Naval Affairs 
and related to me as an individual well into my own career. 

Instructor's attitude toward students, i.e., was respectful of their 
opinions and questions. 

Making the student feel stupid; **That’s so trivial I can't spend time 
on that." 
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Talking down to the students when most of the class had more practical 
experience in the matter than the professor. 

He spent time with students rather than research. He took a personal 
interest in each student in the class. 

Talking down to the class. 



INSTRUCTOR ATTITUDE TOWARD STUDENT QUESTIONS AND CONTENT 
OF STUDENT UNDERSTANDING 



He stressed important points which we could expect to see in higher 
level studies and was concerned with building good foundations. 

He fielded questions without making the student feel like an idiot 
for asking the question. 

Questions raised merely brought attacks on the questioner as to the 
student’s inability to handle the particular problem. 

Never-mind-how-it-*works“just-p lug-in- the~numbers type attitude. 

/ 

He became upset, even angry, if students questioned his methods. 

The instructor was concerned about concepts and understanding them 
rather than overwhelming students with insignificant dates and unrelated 
facts. 

His complete disregard for the questions that were asked and his failure 
to answer them . 



STUDENT COMPETENCE GAIN 



My most vivid recollection of instructor satisfaction was after the 
completion of a particular course. On reflection, I realized that the 
instructor had given us a conceptual approach to problem solving rather 
than a mechanical ’’plug and chug" approach that is common to the majority 
of such classes. 

I felt at the conclusion of the course that I had learned something 
and that it was useful. 

Satisfaction is coupled with the feeling that something is being 
obtained from the course. 

When, after six months, you can’t remember anything about the course . . . 
We finished no smarter than when we started. 
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A feeling of competence in the subject after the course. 

Most of the information I gained from this course was on my own from 
reading and outside help from friends. I felt I could have missed 75% 
of the class meetings and learned as much. 



INSTRUCTOR ABILITY TO CONTROL CLASS TIME 



The effectiveness he demonstrated in encouraging class participation. 

He did not try to cram 80 minutes of lecture in a 50 minute time frame. 
He left time to clear up on-the-spot questions. 

Wasted no time. If he didn^t have any more lecture material, he stopped 
lecturing and class was over. 



INSTRUCTOR AVAILABILITY 



He was never too busy to talk to a student and did not give the appear- 
ance of being bothered or held up — in fact, he seemed to like to talk 
to you if you went to see him with a question. 

He was available for consultation at almost anytime outside class, 
rather than only during office hours. 

He was available at any time to go over difficult areas with students. 

If it was good golf weather his door was locked. If the weather was 
poor, it was closed. He kept us informed of his scores. 



INSTRUCTOR ABILITY TO EVOKE INTEREST 



The instructor assigned readings from the required text and then every 
lecture hour repeated, often verbatim, the material from the text. It 
is difficult to listen to an instructor re-read to you for an hour 
the material you read the night before. 

He made his presentations interesting, if not downright entertaining, 
by keeping them relevant, by using real world situations as examples, 
and by a highly developed sense of humor. 

Attempt on part of the instructor to make even the most unappealing 
part of the course interesting. 

Taught from lesson plans and book. Instructor knew equation and figure 
numbers **by heart" ... he had taught the course so many times it had 
very little appeal. 

He used interesting examples to illustrate his topic. 
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